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Abstract 

The  initial  aim  of  this  project  was  to  combine  (1)  wavelet-based  image  quality  enhancement 
techniques  that  have  recently  been  introduced  in  the  literature  with  (2)  our  previously 
developed  color  remapping  technique  to  enhance  the  articulation  of  details  in  color  fused 
multiband  nighttime  imagery. 

After  implementing  several  of  the  wavelet  super-resolution  techniques,  we  observed  that  their 
performance  was  not  as  good  as  claimed  by  the  authors.  Although  processed  images 
sometimes  looked  sharper,  the  overall  effect  was  minimal.  Moreover,  the  methods  did  not 
work  on  all  images  tested.  After  consulting  mathematicians  and  after  extensive  discussions 
with  Dr.  Pinkus  and  Mr.  Dommett  (Wright-Patterson  AFB,  OH)  we  decided  not  to  pursue 
this  approach. 

In  consultation  with  Dr.  Pinkus,  we  identified  and  decided  to  investigate  the  following  related 
research  topics: 

1 .  The  lack  of  publicly  available  registered  dynamic  multiband  imagery  forms  a  serious 
drawback  for  the  development  and  optimization  of  dynamic  image  fusion  algorithms. 

2.  Guided  filters  appear  promising  candidates  to  replace  standard  low-pass  and  band¬ 
pass  filters  in  multiscale  image  fusion  schemes. 

3.  Color  grading  may  give  multispectral  night  vision  imagery  a  more  realistic  color 
appearance. 

Each  of  these  three  topics  is  related  to  the  original  research  topic,  since  they  all  concern  the 
enhancement  of  details  in  colored  fused  imagery  for  visual  inspection. 

In  response  to  topic  1,  we  collected  a  large  database  of  dynamic  imagery  that  was  collected 
previously  during  several  previous  field  trials  with  the  TNO  TRICLOBS  (TRI-band  Color 
Low-light  OBServation)  all-day  all-weather  surveillance  system.  We  deposited  this  database 
in  the  Figshare  open  repository  (https://figshare.com)  and  we  wrote  a  paper  for  an  open 
access  journal  (PLoS  One)  describing  the  dataset.  Sharing  this  dataset  with  the  research 
community  will  enable  the  development  and  evaluation  of  (both  static  and  dynamic)  image 
fusion,  enhancement  and  color  mapping  algorithms.  Publishing  the  dataset  in  an  open  access 
journal  will  give  it  wide  exposure. 

In  response  to  topic  2  we  developed  a  multiresolution  image  fusion  scheme  based  on  guided 
filtering.  The  scheme  combines  state-of-the-art  performance  with  a  simple  implementation 
and  computational  efficiency.  In  addition,  we  developed  a  new  Alternating  Guided  Filter, 
which  exactly  preserves  the  edges  of  large-scale  details  (no  blurring  or  distortion)  while 
eliminating  fine-scale  details. 

Finally,  in  response  to  topic  3,  we  developed  an  improved  color  remapping  scheme  that  gives 
multiband  false  color  images  a  more  realistic  natural  daytime  color  appearance. 
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1  Original  aim  of  the  project 


Recently  techniques  were  introduced  that  claimed  to  yield  “super  resolved”  up-scaled  images 
by  combining  standard  interpolation  techniques  with  the  wavelet  transform  (Kumar  & 
Nagaraj,  2013,  Tripathi  &  Kirar,  2014).  In  previous  studies  we  developed  a  color  remapping 
technique  that  gives  multiband  nighttime  imagery  an  intuitive  and  stable  color  appearance 
(Hogervorst  &  Toet,  2010).  We  showed  that  this  color  remapping  procedure  enhances  scene 
perception  compared  to  conventional  monochromatic  nighttime  imagery,  and  may  be 
deployed  to  drive  human  fixation  behavior  (Toet,  et  al.,  2014).  The  original  aim  of  this 
project  was  to  combine  the  abovementioned  wavelet-based  image  quality  enhancement 
techniques  with  our  color  remapping  technique  to  further  enhance  the  articulation  of  details 
in  color  fused  multiband  nighttime  imagery. 


2  Initial  results  on  wavelet  super- resolution 

We  implemented  several  of  the  wavelet  super-resolution  techniques  published  in  the 
literature  (see  Kumar  &  Nagaraj,  2013,  Tripathi  &  Kirar,  2014).  However,  the  results  we 
obtained  were  mostly  disappointing.  Although  images  sometimes  looked  somewhat  sharper, 
the  overall  effect  was  minimal.  Moreover,  the  methods  did  not  work  on  all  images  tested. 

We  discussed  these  initial  findings  with  Dr.  Pinkus  and  Mr.  Dommett  and  we  provided  them 
with  our  code  for  evaluation.  They  agreed  with  our  conclusion  that  the  claim  that  these 
wavelet  based  algorithms  are  capable  of  enhancing  image  resolution  cannot  be  substantiated. 

In  addition,  we  consulted  two  mathematicians  from  the  Centrum  Wiskunde  &  Informatica  in 
Amsterdam  (https://www.cwi.nl),  who  checked  the  algorithms  and  agreed  with  our  findings 
that  these  algorithms  do  not  effectively  enhance  image  resolution.  In  addition,  they  concluded 
that  there  is  no  mathematical  foundational  proof  that  these  algorithms  should  have  a  positive 
effect  in  all  general  images. 

Although  these  algorithms  can  sometimes  make  an  image  look  "sharper"  they  do  not  increase 
resolution.  As  Mr.  Dommett  noted:  “The  claims  of  these  types  of  algorithms  contradict  the 
Nyquist  Sampling  Theorem  -  they  imply  that  you  can  represent  a  signal  (image)  with  a  lower 
sampling  frequency  and  then  you  can  generate  more  information  by  re-arranging  those 
numbers,  applying  "some"  manipulation,  and  then  re-arranging  the  numbers  back  into  an 
image.  At  best,  the  new  data  is  a  "guess".  Without  a  priori  knowledge  of  the  scene,  lighting 
and  the  image  capturing  device,  the  chosen  manipulations  are  no  more  valid  than 
interpolation.  My  conclusion  is  that  these  algorithms  offer  no  substantial  improvement  over 
(for  example)  bicubic  interpolation  with  post-sharpening. “ 

Hence,  based  on  these  initial  findings,  and  after  consulting  Dr.  Pinkus,  we  decided  not  to 
pursue  this  line  of  research  any  further. 

After  extensive  discussions  with  Dr.  Pinkus  we  decided  to  continue  the  project  with  the 
following  actions: 

1.  To  collect  a  large  set  of  registered  dynamic  multiband  imagery,  recorded  during 
previous  field  trials  with  our  TRICLOBS  multiband  camera  system,  make  this  set 
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available  to  the  scientific  community  through  an  open  repository,  and  publish  a  paper 
describing  this  dataset  in  an  open  access  journal. 

2.  To  integrate  recently  introduced  guided  filtering  techniques  in  a  multiresolution  image 
fusion  scheme. 

3.  To  develop  improved  color  mapping  algorithms  to  give  multiband  nighttime  imagery 
a  more  realistic  appearance. 

The  results  of  these  investigations  will  be  presented  in  the  next  sections. 

3  Dynamic  Multi-band  Image  Data  Set 

The  fusion  and  enhancement  of  multiband  nighttime  imagery  for  surveillance  and  navigation 
has  been  the  subject  of  extensive  research  for  over  two  decades.  Despite  the  ongoing  efforts 
in  this  area  there  is  still  only  a  small  number  of  static  multiband  test  images  available  for  the 
development  and  evaluation  of  new  image  fusion  and  enhancement  methods.  Moreover, 
dynamic  multiband  imagery  is  also  currently  lacking.  To  fill  this  gap  we  present  the 
TRICLOBS  dynamic  multi-band  image  data  set  containing  sixteen  registered  visual  (0.4  - 
0.7pm),  near-infrared  (NIR,  0.7  -  1.0pm)  and  long-wave  infrared  (LWIR,  8  -  14pm)  motion 
sequences.  They  represent  different  military  and  civilian  surveillance  scenarios  registered  in 
three  different  scenes.  Scenes  include  (military  and  civilian)  people  that  are  stationary, 
walking  or  running,  or  carrying  various  objects.  Vehicles,  foliage,  and  buildings  or  other 
man-made  structures  are  also  included  in  the  scenes.  This  data  set  is  primarily  intended  for 
the  development  and  evaluation  of  image  fusion,  enhancement  and  color  mapping  algorithms 
for  short-range  surveillance  applications. 

The  imagery  was  collected  during  several  previous  field  trials  with  the  TRICLOBS  (TRI- 
band  Color  Low-light  OBServation)  all-day  all-weather  surveillance  system,  that  was 
constructed  in  the  course  of  the  previous  EOARD-funded  project  “Registration  of  a  Dynamic 
Multimodal  Target  Image  Test  Set  for  the  Evaluation  of  Image  Fusion  Techniques”  (Grant 
FA8655-1 1-1-3015;  see  e.g.  Dijk  et  al.,  2010;  Toet  et  al.,  2010;  Toet  et  al.,  2011;  Toet  & 
Hogervorst,  2009;  Toet  &  Hogervorst,  2010).This  system  registers  a  scene  in  the  Visual,  NIR 
and  LWIR  part  of  the  electromagnetic  spectrum  using  three  optically  aligned  sensors  (two 
digital  image  intensifiers  and  an  uncooled  long-wave  infrared  microbolometer).  The  three 
sensor  signals  are  mapped  to  three  individual  RGB  color  channels,  digitized,  and  stored  as 
uncompressed  RGB  (false)  color  frames. 

We  will  make  the  TRICLOBS  data  publicly  available  through  the  Figshare  open  repository 
(https://figshare.com/).  Sharing  this  dataset  with  the  research  community  will  enable  the 
development  and  evaluation  of  (both  static  and  dynamic)  image  fusion,  enhancement  and 
color  mapping  algorithms.  To  allow  the  development  of  realistic  color  remapping  procedures, 
the  data  set  also  contains  color  photographs  of  each  of  the  three  scenes.  The  color  statistics 
derived  from  these  photographs  can  be  used  to  define  color  mappings  that  give  the  multi¬ 
band  imagery  a  realistic  color  appearance. 


Reference  c.q.  deliverable 

Toet,  A.,  Hogervorst,  M.A.,  &  Pinkus,  A.R.  (submitted).  The  TRICLOBS  Dynamic  Multi¬ 
band  Image  Data  Set  for  the  Development  and  Evaluation  of  Image  Fusion  Methods.  PLoS 
One. 
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4  Multiscale  Image  Fusion  through  Guided  Filtering 


We  developed  a  multiresolution  image  fusion  scheme  based  on  guided  filtering.  Guided 
filtering  can  effectively  reduce  noise  while  preserving  detail  boundaries.  When  applied  in  an 
iterative  mode,  guided  filtering  selectively  eliminates  small  scale  details  while  restoring 
larger  scale  edges.  The  proposed  multi-scale  image  fusion  scheme  achieves  spatial 
consistency  by  using  guided  filtering  both  at  the  decomposition  and  at  the  recombination 
stage  of  the  multiscale  fusion  process.  First,  size- selective  iterative  guided  filtering  is  applied 
to  decompose  the  source  images  into  base  and  detail  layers  at  multiple  levels  of  resolution. 
Then,  frequency-tuned  filtering  is  used  to  compute  saliency  maps  at  successive  levels  of 
resolution.  Next,  at  each  resolution  level  binary  weighting  maps  are  obtained  as  the 
pixelwise  maximum  of  corresponding  source  saliency  maps.  Guided  filtering  of  the  binary 
weighting  maps  with  their  corresponding  source  images  as  guidance  images  serves  to  reduce 
noise  and  to  restore  spatial  consistency.  The  final  fused  image  is  obtained  as  the  weighted 
recombination  of  the  individual  detail  layers  and  the  mean  of  the  lowest  resolution  base 
layers.  Application  to  multiband  visual  (intensified)  and  thermal  infrared  imagery 
demonstrates  that  the  proposed  method  obtains  state-of-the-art  performance  for  the  fusion  of 
multi  spectral  night  vision  images.  The  method  has  a  simple  implementation  and  is 
computationally  efficient. 


References  c.q.  deliverables 

Toet,  A.  (2016b).  Iterative  Guided  Image  Fusion.  PeerJ  Computer  Science,  2  (e80),  1-26.  doi: 
10.77 17/peerj-cs. 80.  https://peeri.com/articles/cs-80/ 

Toet,  A.,  &  Hogervorst,  M.A.  (2016).  Multiscale  Image  Fusion  through  Guided  Filtering. 
Target  and  Background  Signatures,  SPIE-9997.  In  preparation. 


5  Alternating  Guided  Image  Filtering 

Edge  preserving  filters  aim  to  simplify  the  representation  of  images  (e.g.,  by  reducing  noise 
or  eliminating  irrelevant  detail)  while  preserving  their  most  significant  edges.  These  filters 
are  typically  nonlinear  and  locally  smooth  the  image  structure  while  minimizing  both  blurring 
and  over-sharpening  of  visually  important  edges.  We  developed  the  new  Alternating  Guided 
Filter  (AGF)  that  achieves  edge  preserving  smoothing  by  combining  two  recently  introduced 
filters:  the  Rolling  Guided  Filter  (RGF)  and  the  Smooth  and  iteratively  Restore  Filter  (SiR). 
We  showed  that  the  integration  of  RGF  and  SiR  in  an  alternating  iterative  framework  results 
in  a  new  smoothing  operator  that  preserves  significant  image  edges  while  effectively 
eliminating  small  scale  details.  The  AGF  combines  the  large  scale  edge  and  local  intensity 
preserving  properties  of  the  RGF  with  the  edge  restoring  properties  of  the  SiR  while 
eliminating  the  drawbacks  of  both  previous  methods  (i.e.,  edge  curvature  smoothing  by  RGF 
and  local  intensity  reduction  and  restoration  of  small  scale  details  near  large  scale  edges  by 
SiR).  The  AGF  is  simple  to  implement  and  efficient,  and  produces  high-quality  results.  We 
demonstrated  the  effectiveness  of  AGF  on  a  variety  of  images,  and  provided  a  public  code  to 
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facilitate  future  studies.  Applying  AGF  in  a  multiresolution  framework  may  lead  to  improved 
image  fusion  schemes. 

Reference  c.q.  deliverable 

Toet,  A.  (2016a).  Alternating  Guided  Image  Filtering.  PeerJ  Computer  Science,  2,  e72.  doi: 
10.77 17/peerj-cs. 72.  https://peeri.com/articles/cs-72/ 


6  Coloring  multiband  night  vision  images 

Previously,  we  presented  a  method  for  applying  daytime  colors  to  fused  nighttime  (e.g. 
intensified  and  LWIR)  imagery  (Toet  &  Hogervorst,  2012).  Our  color  mapping  not  only 
imparts  a  natural  daylight  appearance  to  multiband  nighttime  images  but  also  enhances  the 
contrast  and  visibility  of  otherwise  obscure  detail.  As  a  result  this  colorizing  method  leads  to 
increased  ease  of  interpretation,  better  discrimination  and  identification  of  materials,  faster 
reaction  times  and  ultimately  improved  situational  awareness  (Toet  et  al.,  2014).  A  crucial 
step  in  this  coloring  process  is  the  choice  of  a  suitable  color  mapping  scheme.  When  daytime 
color  images  and  multiband  sensor  images  of  the  same  scene  are  available  the  color  mapping 
can  be  derived  from  matching  image  samples  (i.e.,  by  relating  color  values  to  sensor  signal 
intensities).  When  no  exact  matching  reference  images  are  available  the  color  transformation 
can  be  derived  from  the  first-order  statistical  properties  of  the  reference  image  and  the 
multiband  sensor  image  (Toet,  2003).  We  therefore  investigated  various  color  grading 
methods  to  implement  this  step.  We  developed  an  improved  color  grading  scheme  leading  to 
colors  that  better  match  the  natural  daytime  colors. 

Reference  (c.q.  deliverable) 

Hogervorst,  M.A.,  &  Toet,  A.  (2016).  Improved  Colour  Matching  Technique  for  Fused 
Nighttime  Imagery  with  Daytime  Colours.  Electro-Optical  and  Infrared  Systems:  Technology 
and  Applications,  SPIE-9987:  SPIE.  In  press. 
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